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Abstract 

Seattle University recently decided to replace three 
separate, computerized student-information systems with a single, 
integrated system. The complexity of this decision was managed 
with a multicriteria method which was used to evaluate 
alterr. ^ive systems. The method took into account the many and 
sometimes conflicting concerns, of the people who would use 
whatever system was finally selected. Multicriteria analysis 
not only provided a way of managing a large amount of 
information, but apparently reduced people's resistance to 
change. The method is simple in structure and can be adapted to 
different kinds of decisions and decision-making processes 
encountered by institutional researchers. 
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Introduction 

The purpose of this paper is to demonstrate how an easy-to- 
understand multicriteria method can be used to manage a large 
amount of information about the consequences of alternative 
courses of action. We illustrate how the method was used to aid 
in making a complex decision at Seattle University: the 
selection of a computerized student-information system. The 
method, however, can be used to assess the relative 
attractiveness of alternative ways of accomplishing virtually any 
specified ends. We do not discuss the relative merits of 
various computerized student-information systems. Our focus is 
on the method of assessment and how the method facilitated the 
decision-making process. 

To begin, we present the background of the decision 
situation, then explain the multicriteria method that was used to 
aid in reaching a decision, and finally discuss the validity of 
the method as a decision«.aiding tool. 

Background 

In the fall of 1986, an implementation task force at 
Seattle University (SU) had been working for a year to phase in a 
computerized student-information system called ISIS (Integrated 
Student Information System). ISIS was to replace independent 
systems in the offices of admissions, financial aid, the 
registrar, and the controller. Collectively, the members of the 
task force represented all of the SU staff members who would 
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eventually be using ISIS* After a year*s preparation to phase in 
ISIS and phase out the various existing systems, members of the 
task force began to question if ISIS would be able to perform all 
necessary functions adequately. The foremost a'^eas of concern 
were; (1) What student-service policies would have to be changed 
to make ISIS work? (2) How many hours of software modification 
would be required? and (3) Would important capabilities of the 
existing systems be lost, such as; 
o Ad hoc reporting? 

o Independence from data-processing personnel for routine 
operations? 

o Flexibility in building and maintaining displays of student 
data? 

o On-line inquiry about individual students? 
o Production of enrollment statistics on-line? 
o Production of letters and selection of mailing labels? 
In addition to the above, other concerns arose: 
o The scope and cost of the modifications to be made' to ISIS 

were greater than anticipated; 
o The original implementation schedule could not be met; and 
o The process that was being used to develop and review 

modifications to ISIS was creating conflict between ISIS 

users and the programmers who would maintain and modify 

ISIS. 

Because of these concerns, SU*s Cabinet (the president and 
vice presidents) authorized the formation of the ISIS Study 
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Group (ISG) to reevaluate the wisdom of replacing functioning 
information systems with ISIS* The authors of this paper and one 
other person were the members of the ISG. 

The ISG*s first task was to examine the process that had led 
to the decision to purchase the ISIS system and had guided the 
work of the ISIS-implementat ion task force during the preceding 
year. The ISG understood its role to include allowing SU*s 
computer users opportunities to express their concerns about the 
adequacy of ISIS and to suggest alternatives. To facilitate 
communication between the ISG and computer users, an ad hoc 
advisory group was formed. This group included the already 
constituted ISIS-implementat ion task force and the member of SU*s 
Administrative Computer Users Committee, an advisory committee to 
the department of Computer and Information Services. This 
committee is made up of representatives of all administrative 
users. Potential users from the academic ranks were also asked 
to serve, including the deans and a representative from the 
Faculty Senate. In all, the Ad Hoc Advisory Group had 25 
members. 

The ISIS Study Group began its work by reviewing the reports 
and memoranda that had led to the decision to replace existing 
systems with ISIS. The ISG summarized its findings in statements 
of fact. These summaries, along with the documents themselves, 
were sent to members of the Ad Hoc Advisory Group for their 
review and comment. This process allowed the ISG to focus on the 
concerns about ISIS and develop evaluation criteria based on 
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these concerns. These criteria were then used to evaluate three 
alternative courses of action, which are dv^scribed in the next 
section. The ISG had also considered several other student- 
information systems but found them to be too big or too small for 
SU or incompatible with the hardware SU had available. And 
purchasing new hardware was not financially feasible. 

Description of Alternatives 
Three alternatives for meeting SU*s student-information 
requirements were selected for evaluation: 

(1) Continue using all currently functioning student- 
information systems — four independent systems serving the 
offices of admissions, financial aid, the registrar, and the 
controller. All of these systems ran on SU*s Hewlett- 
Packard ( HP) computer . 

(2) Install ISIS with the modifications that had already 
been made or had been approved by SU*s Cabinet. This 
version of ISIS would run on 3U*s newly purchased IBM 
computer. 

(3) Install ISIS with all the modifications that had been 
recommended six months earlier by the ISIS-implementation 
task force. (The task force had recommended many more hours 
of modification than had been approved by the Cabinet at 
that point.) This version of ISIS would also run on SU»s 
newly purchased IBM computer. 
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Evaluating the Alternatives 
To evaluate the relative adequacy of the three alternative 
information systems, the ISG specified measurable criteria on a 
draft of the evaluation form that was eventually used. These 
criteria emanated from the users' concerns. Computer users from the 
offices of admissions, financial aid, housing, the registrar, and 
the controller were then asked to review all the criteria, weigh the 
importance of each criterion, and rate the alternatives on the 
criteria that related to their concerns. The rating scales used on 
the evaluation form are simple (see Figure 1). For example, the 
degree to which various functional requirements were judged to be 
met by a particular alternative is rated on a three point scale: 
1=does not meet, 2=meets minimum, and 3=exceeds. This and the other 
simple scales on the evaluation form made sense to the users because 
the users were well familiar with all three alternatives.> 

The ISG assigned weights to the criteria based on a composite 
of the weights suggested by the individual users, then scored each 
alternative on each criterion. Decisions about weights and scores 
were reached by consensus after ISG members sought clarification 
from each other and, in several cases, after they consulted users. 

To derive total scores for alternatives, numerical scores were 
assigned to each alternative on each criterion. Then scores for 
each criterion were multiplied by the assigned weight for that 
criterion. Finally, weighted scores for each alternative were 
summed. ISIS with all requested modifications received the highest 

ERJC 



Evaluation Criteria 



The weight (relative importance) of each criterion 
appears in parentheses: low=l, medium=2, high=3. 



1 . Degree to which meets functional 
requirements for: 

a* Course catalogs (course master in HP) 

b* Class schedule 

c • Recruitment/admissions 

d. Financial aid~IMAS/PARS 

e* Financial-aid awards 

fv Student demographics 

g* Enrollment 

h. Housing 

i» Accounts receivable 

j* Grades/academic history 

k» Degree audit 

1* Graduation tracking 

m. Selection and production of labels 

n. Flexibility of selection and 
production of letters 

o* Assessment of fees 

p» Term structure 

q* Student ID 

r* Ad hoc reporting 

2» Degree to which software is user 
friendly: 

a» Adequacy of screen design 

b» Ease of data entry/correction 

c. Ease of sign-on and sign-off 

r-ni/^"*'® Multicriteria evaluation form 
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3. Degree to which software is flexible: 
a* Ability to adapt systems tables 

b. Deg' of departmental-lerel 
security 

c. Degree to which allows user 
initiation of reports 

Degree of compat-'billty of printed 
output with existing printed output: 

a» Format of transcripts 

b» Report dimensions 

c* Report design 

5. Degree to which software is compatible 
with current SU student-services policies 

6. Degree of availability to user population 

7. Affect on morale 



8. Degree to which normal operations 
will be disrupted 

9. Degree to which staff workload 
will be increased 

10. Degree of additional training required 



11. Availability of current and future 
computer systems: 

a. Are programmer-productivity tools 
available? 

b. Are user-query tools available? 

c. Are the hardware and systems software 
commonly used in higher education? 

d. Is the hardware/software environment 
served by several higher-education 
software vendors? 
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. e* Will new hardware and systems 
software be available? 

12. Degree to which meets technical standard 

a* Is documented — user, programmer, 
operations 

b» Uses programming standards 

c. Is easily modified 

d* Uses a proven data-base management 
system 

13* Degree to which meets data standards: 

a. Ease of integration 

b. Provides audit trails of activity 
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score and the existing HP system received the lowest • We will not, 
however, elaborate further on the ratings assigned to the 
alternatives, since our purpose is to describe the evaluation 
process,, not to support one of the alternatives. 

When the ISG completed the evaluation form with weights, 
ratings, and total scores for the alternatives, the ISIS Ad Hoc 
Advisory Group was called together to review and discuss the 
results of the ISG*s numerical evaluation. Presentation of the 
results on the evaluation form allowed a large number of people with 
different concerns and perspectives to focus on a complex body of 
information. Further, the multicriteria method allowed everyone a 
chance to see that his or her concerns were explicitly taken into 
account in evaluating the alternatives. 

Results of the ISG*s systematic and comprehensive evaluation 
pointed to ISIS with all modifications as the preferable 
alternative. During the discussion of these results, users' 
objections to replacing existing systems with ISIS seemed to 
dissolve. Consequently, SU administrators decided to install ISIS 
with most of the modifications requested by the implementation task 
force. 

Validity of the Method 
While the specific alternative selected was certainly important 
to the people at SU, our purpose is not to focus on the merits of 
the alternative selected but on the method used to facilitate the 
decision-making process. 

Id 
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The criteria the ISG developed reflected the concerns and 
judgments of the particular set of actors at SU. Similar concerns 
might exist at another university with different actors, but it*s 
not likely these concerns would be identical to the concerns at SU. 
The value of the method, then, is not in its ability to identify the 
optimal choice in an objective sense. The value of the method is 
in the structure it provides for weighing a large number of complex 
and interrelated concerns. Multicriteria analysis allows the 
comprehensive, balanced consideration of the many issues that arise 
in making complex decisions. 

The method we used is not new, but an adaptation of a class of 
methods well documented in the evaluation literature in the field of 
urban planning. In particular, multicriteria analysis is an 
adaptation of Morris Hill^s (1968) goals-achievement matrix. An 
application of Hill^s method ?s illustrated by Miller (1980). For 
in-depth treatments of multicriteria methods, see Voogd (1983) and 
Faludi and Voogd (1985). 

While multicriteria methods are well known in the field of 
urban planning, apparently they are little known, or at least not 
often used, in the field of higher education. This conclusion is 
based on the fact that a computer-assisted search of the U.S. 
Department of Education's bibliographic data base — Educational 
Resources Information Center (ERIC) — revealed no applications of 
multicriteria methods of the type we used. Our method appears, 
however, to embody the concept of multiple-perspective evaluation 
described by Palola and Bradley (1976) in a paper discovered in our 
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search of ERIC. That is, the multicriteria method we used, in 
effect, allowed the selection of a student-information system to be 
considered from multiple perspectives. This occurred because the 
concerns of all interested parties were translated into one or more 
of the criteria that appear on the evaluation form. 

In uaing multicriteria analysis, if a particular concern is 
not represen'jed by one of the criteria, its absence will likely 
Jump out at the people who have the concern. They can then request 
that appropriate criteria be added to take their concern into 
account. Likewise, if a group feels that particular criteria have 
been given insufficient weight, or too much weight, weights can be 
changed. (There is no theoretical limit on the magnitude of a 
weight.) Adjustments to criteria and weights happen easily and 
naturally as people are asked to review drafts of the evaluation 
form. In this sense, the use of the method is self correcting. 

The value of using many criteria lies in the fact that when all 
the concerns are "on the table," no one of them, no matter how 
important to a particular group, is likely to be given undue weight 
in the final decision. This characteristic of multicriteria 
analysis can eliminate resistance to change on the part of a group 
that is dissatisfied with one or a few features of an alternative: 
When members of the objecting group are given a way of viewing the 
alternative from the perspective of all the criteria, they may 
decide it is the preferable choice after all. In this case, one 
group of users had decided that the existing HP system was 
preferable to either of the ISIS alternatives. When, however, their 
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concerns were taken into account explicitly in the form of criteria 
and weights, along with many other concerns, they accepted ISIS with 
all modifications. 

Conclusion 

Multicriteria analysis, used in the way we have described, 
forces interested parties to advocate their concerns in light of 
everyone else^s concerns. The process we used allowed the parties 
to see the decision from multiple perspectives and move to 
consensus. While we do not claim this will always occur, the 
outcome in this case suggests that multicriteria analysis can, at 
the very least, facilitate movement toward consensus. 

We found multicriteria analysis to be a useful, easily 
understood tool for managing large amounts of information. Based on 
our experience with and knowledge of the tool, we feel it can, and 
should, be adapted to different kinds of decisions and decision- 
making processes encountered by institutional researchers. 
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